Large corporations have for many years been concentrating their compute resources in data centers. This trend has accelerated over the last few years as server virtualization technology has become more and more prevalent. As data centers have become larger, some data center operators have begun to offer computing, storage, and network communication resources to outside customers. The offered services typically consist of elastic, on demand processing, storage that for most practical purposes is limited only by the customer's ability to pay, and network bandwidth into the Internet. This development is called cloud computing.
Server virtualization technology allows a pool of servers to be managed as essentially one large computer resource. A layer of software called a hypervisor sits between the operating system and the hardware. The hypervisor schedules the execution of virtual machines (“VMs”) on a virtualized server. A VM is an operating system image packaged with some applications. The hypervisor allows a VM to be suspended and moved between servers to load balance. Load balancing and monitoring of VM execution to catch crashes provides the same kind of fault tolerance and scalability services for enterprise applications that are achieved at much higher cost with specialized solutions. A cloud manager system oversees the execution of VMs; scheduling execution to meet demand, to optimize server utilization, and to minimize power consumption. The cloud execution manager can schedule execution to allow in-service upgrade of hardware and software without impacting ongoing service provision.
In order to support arbitrary movement of VMs between machines, the networking within the data center must also be virtualized. Most clouds today virtualize the network by incorporating a virtual switch into the hypervisor. The virtual switch provides virtual network ports to the VMs executing under the control of the hypervisor. The virtual switch software also allows the network resources to be virtualized in a manner similar to how the server resources are virtualized by the hypervisor. The hypervisor and the virtual switch can thereby cooperate to allow VMs to be moved between servers. When the hypervisor moves a VM, it communicates with the virtual switch about the new location, and the virtual switch ensures that the network routing tables for the VM's addresses (layer 2 Media Access Control (“MAC”) address, potentially also the internet protocol (“IP”) address) are updated so packets are routed to the new location.
Many cloud computing facilities only support Web services applications. Web services applications consist of a load balancing front end that dispatches requests to a pool of Web servers. The requests originate conceptually from applications on the Internet and therefore the security and privacy requirements are much looser than for applications in a private corporate network. A newer trend is secure multi-tenancy, in which the cloud provider offers virtual private network (“VPN”) like connections between the client's distributed office networks outside the cloud and a VPN within the cloud. This allows the client's applications within the cloud to operate in a network environment that resembles a corporate wide area network (“WAN”). For private data centers, in which services are only offered to customers within the corporation owning the data center, the security and privacy requirements for multi-tenancy are relaxed. For public data centers, the cloud operator must ensure that the traffic from multiple tenants is isolated and there is no possibility for traffic from one client to reach another. In either case, cloud computing facilities tend to implement cloud computer networks using MAC layer virtual local area networks (“VLANs”).
For example, two Virtual Private Clouds (“VPCs”) can be set up for two different external enterprise customers. A VPC consists of a collection of VMs, storage, and networking resources that provide secure multi-tenancy to the enterprises renting space in the cloud. The enterprise customers connect into the VPCs via VPNs over the Internet running on a public operator network.
In order to add a new service instance (a new VM) to a VPC, a cloud execution manager initializes the VM to run on a hypervisor on a virtualized server. The virtual switch on the virtualized server is configured to include the VM in a VLAN that is part of the VPN for the enterprise adding the new VM. In some cases, a virtual customer edge router is updated for the new service and a provider edge router in the cloud computing facility is updated with the new service.
In order to provide VPNs, cloud computing facilities implement one of three solutions. First, each tenant receives a separate VLAN slice. Second, tenant VPNs are implemented using IP encapsulation. Third, tenant VPNs are implemented using MAC address encapsulation. Each of these solutions suffers from deficiencies.
If the cloud uses VLAN isolation, each tenant is assigned a separate VLAN tag and the cloud network is run as a flat Layer 2 network. The VLAN tag has 12 bits, so if a cloud operator uses VLAN isolation the number of tenants is restricted to 4096. This limit provides a major limitation.
Another problem with VLAN isolation is that standard area networking (e.g., LAN, WAN, MAN; Institute of Electrical and Electronics Engineers (“IEEE”) 802.1) switching uses the spanning tree protocol (“STP”) to set up routes. In order to remove the possibility of routing loops, STP designates one and only one path between a source address and a destination address, regardless of whether there are multiple routes. This can lead to congestion and underutilization of the switching fabric when the spanning tree route comes under traffic pressure and alternate routes are neglected.
With IP encapsulation, the cloud is run as a routed IP network and IP tunnels are used to isolate tenant's traffic. The traffic from a tenant is encapsulated in an IP packet (typically using Generic Routing Encapsulation (“GRE”)) with the endpoints of the tunnel being the source and destination virtual switches on the source and destination virtualized servers where the VMs are running.
IP encapsulation allows a client to define an arbitrary layer 2 service on top by encapsulating Ethernet frames in Layer 2 Tunneling Protocol (“L2TP”). It also allows large numbers of tenants, constrained only by the cloud-wide IP address space. The tenant can also deploy their own IP address space on top of the IP tunnels. However, without other measures, IP routing also selects a single route and so multipath routes are neglected, leading to unnecessary congestion. The cloud routing can utilize Equal Cost Multipath to spread packets over multiple links but at the cost of additional configuration complexity.
In general, configuring an IP routed network is time consuming and routers tend to be more expensive devices than simple switches. In addition, IP networks have limited means to provision dedicated bandwidth, which might be necessary for large data flows.
With MAC encapsulation, the separate tenant VPNs are run as encapsulated inside MAC tunnels, similar to IP encapsulation. The endpoints of the MAC tunnels are typically the virtual switches on the source and destination virtualized servers where the VMs are running.
MAC encapsulation offers similar benefits to IP encapsulation over VLAN isolation, with the added benefit that the cloud can be run as a flat Layer 2 network if desired. The disadvantage is that there are few standards for the signaling protocols for MAC encapsulation, unlike IP, and, although there are standards for data plane encapsulation, they are not used in some existing cloud management software products. This runs the risk of not working with certain types of hardware. Configuring and maintaining a MAC encapsulation network is also more complex than maintaining a VLAN isolated network.